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Background / Context: 

Description of prior research and its intellectual context. 

Contradictory findings from well-implemented rigorous evaluations invite us to identify the 
differences that might explain the contradictions, helping us generate testable hypotheses for new 
research. However, poorly designed and implemented studies do little to move our 
understanding forward. This panel will examine efforts to ensure that the large number of local 
evaluations being conducted as part of four federally funded grant programs generate rigorous 
findings that can inform our understanding of contradictory findings. 

The Department of Education (ED), the Department of Labor (DOL), and the Office of Health 
and Human Services (HHS) have funded grant programs to support the implementation and 
evaluation of approaches to solving pressing social problems. The panel will focus on the 
Investing in Innovation Program (i3) and Striving Readers Program (both funded by ED), the 
Workforce Innovation Fund (DOL), and the Teen Pregnancy Prevention Program (HHS). These 
programs have clearly made it a priority to protect the federal investment in rigorous, local 
evaluations; they have each put contracts in place with nationally recognized research firms to 
provide technical assistance to support the local evaluators as they conduct rigorous research. 
Three of the four contracts also support the review of the completed studies against standards of 
evidence, and the summary of the evidence that is generated for the federal agencies, policy 
makers, and the respective research communities. 

Having a large body of high-quality evidence about the effectiveness of interventions aimed at 
solving a social problem is a necessary, but not sufficient condition for examining contradictory 
findings. The panelists will engage each other and the audience in a discussion about how to 
leverage the opportunity these emerging bodies of evidence present for identifying and learning 
from varied and contradictory findings. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

Given that these programs are poised to generate a good deal of rigorous evidence for their 
respective fields, what opportunities do they hold for understanding and learning from 
contradictory findings? Can we anticipate some of the sources of variation and contradiction so 
that we can make the most of these opportunities? 

To address these questions, the panelists will first provide a brief description of the programs 
with which they are working, and provide the audience a sense of the contexts in which the 
interventions and evaluations are being implemented. These presentations will be no more than 
8-10 minutes each, or 30 minutes total. 

The panelists will then participate in a discussion moderated by Dr. Robert Granger, centered on 
the following two questions/topics: 
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1. The four grant programs have distinctly different missions, target populations, and 
audiences. Further, they differ with respect to the degree of similarity among the 
interventions they support. For example, the programs supported by Striving Readers 
were all focused on meeting the needs of struggling adolescent readers whereas the 
program supported by i3 tackle a wide range of educational topics. The Workforce 
Innovation Fund and the Teen Pregnancy Prevention Program both have well defined foci 
but fund interventions that take a range of approaches. How do we define domains of 
interventions within which contradictory findings would be informative? How do these 
grant programs help us think about grouping sets of findings to identify contradictions 
and opportunities for future research? 

2. Learning from contradictions requires understanding differences that might have 
generated the contradictory findings. What could we be measuring about these 
evaluations, the interventions being studied, and the contexts in which they are being 
implemented and evaluated that might be related to the contradictions? 

The panelists and moderator will engage in discussion for approximately 30 minutes, leaving the 
final 30 minutes for open discussion of these topics with the audience. The audience will be 
invited to share their thoughts on how we might leverage these opportunities to learn more from 
contradiction, including practical methods for data capture for future investigations. 

Setting: 

Description of the research location. 

The local evaluations being funded by these programs are being conducted across the country, in 
a wide range of contexts, making it a particularly fruitful opportunity for examining variation 
and contradiction. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

The populations, participants, and subjects also come from a range of contexts, given the breadth 
of the evaluations. If we think of the evaluations themselves as ‘subjects’ in a larger 
examination of variation in findings, they too vary along a number of interesting dimensions. 
These include the size and scope of the intervention being delivered, the size and power of the 
evaluation, the methodology used to examine the effectiveness of the intervention, and the 
contexts in which the evaluations are being conducted. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

The Striving Readers Program was aimed at meeting the needs of struggling adolescent readers. 
The i3 program funds educational innovations aimed at improving student outcomes across the 
pre-k to grade 12 spectrum, and funds approaches aimed at affecting school structures and 
processes, teacher practices, and student attitudes and behaviors. 
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Research Design: 

Description of the research design. 

All of the programs support evaluations aimed at understanding the effectiveness of the 
innovations, using quasi-experimental or experimental methods. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

This of course varies across the evaluations. The Panel will discuss with each other and the 
audience what sorts of data elements would be useful to capture now to allow for the 
examination of variation and contradiction when the findings are ultimately available. 

Findings / Results: 

Description of the main findings with specific details. 

Findings will vary across the evaluations, giving rise to interesting contradictions that provide 
opportunities for learning. 

Conclusions: 

Description of conclusions, recommendations , and limitations based on findings. 

The proposed panel discussion will generate conclusions about how we should think about 
grouping evaluations together to identify contradictions, the possible types of hypotheses we 
might generate to explain these contradictions, and what data we might want to be able to test 
those hypotheses down the road. 
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